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ABSTRACT^ . ... 

" / Thi's year-long study was conducted to ./investigate 
effects of various te.acJier accountability factors on teadiiig ■ ' ' ; 
achievement of first-grdde pupils. Each of fifteen schools was 
assign-ed through randomized blocking prqcedures (blocking fa^ctors ^ ^ 
were reading achievement, 'and percentage of minority students) tp one 
of four levels of ^cc.6untability . Accountability levels were defined 
from low to high ^s follojjs; use of the regular first-grade reading 

^program. without Supple mentatioi^; the regular program plus a . 

"supplementary sys'tem incruding Instriacrional ^objectives, regular 
assessment and, remedial exercisesy and objectives-keyed pupil 
performance. records for each of the fifteen, instructional units in 
the program;" the' second condition plus- either a pacing scliedule or 

-reporting of unit scores to^'the school princifJal; and th^ -second 
condition plu^ both pacing and reporting. A 45-if em criterion test 
based on 'the major objectives'* of the; reading prcgram; was individually, 
administered- at .the ^nd of the school year. An L-tdst for >^f our 
rankings" (accountability levels) by three observations Jmean subtest, 
scores' on three major rea^ding objectives) 'revealed a .Significant 
positive relatignship (p <' •001) between level of teacher ' ^ 

accountability and ^pupil reading achievenient.. ^ (Author) , 
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PUPIL ACHIEVEMENT UNDER VARYING LEVELS OF 
TEACHER ACCOUNTABILITY 

Howard J. Sullivan and Fred C. Niedeirmeyer 



c_ ' ' ^ ABSTRACT ^ / 

This year-long study was coi;:iducted to investigate effects of - 
various teacher accountability factors on reading achievement of first- 
grade pupils. At the beginning of the school year, each of 15 schools 
was assigned /through randomized blocking procedures (blocking factors 
were reading achievement and percentage of minority students) to one of 
four* levels of accountability.' Accountability levels were defined from 
low to high .as' follows : (1) use of the regular f irst-^-grade reading pro- 
gram.'without supplementation, (2) the regular program plus a supplementary 
sys,tem including instructional objectives, regular assessment and re- * 
medial exercises, and objectives-keyed pupil performance records for 
each of the 15 instructional units in the program, (3) condition 2»aboye 
plus either a pacing schedule or reporting of unit scores to the school 
principal, (4) condition 2 plus both pacing and reporting. A 45-item 
criterion test based on the major objectives of the 'reading program was ^ 
individually administered at the end of the school year^ An L-test for 
four rankings (accountability levels) by three observations (mean sub- 
test scores on- three major reading objectiv,es) revealed a significant 
positive relationship (p<.001) between level of teacher accountability 
and ipupil reading achievement* » , 



PUPIL achievement; under varying levels of i' ' ^ J 

TEACHER ACCOUNTABILITY 7 - ' 

Howard J. Sullivan apnd Fred C. Niedermeyer . • 

*» ■ ' . ^ t " . . 

The 'idea of -teacher accountability for prO(!ffi&ing demorlstraited gains 
in piipil acKievement currently .enjoys considerable popularity among 
educators- In general, however, specific procedures for promoting 
accountatfility have neither been i^ell^ defined and testecj n'^or installed^ 
in the schools.' Little, if any, experimental re,search hlis been con- 
ducted to investigate the effects of accountability procedures on pupil 
achievement • 

The present study involved an attempt X-o (1)' identify and opera- 
tionalize a set of factors related to aabountability and (2) empiri-^ 
cally determine ,the relationship between these factors^and learner 
achievement. Accountability factors identified and investigated in the 
study were as follows: 

• Specification Qf what the- teacher is accountable for--t^hat is, 
what objectives must be attained -^and provision of a means for 
regular determination by the teacher of whether the objectives 
are being/attained. '° " i «■ 

• Development of a ^'schedule indicating to the teacher the max- 
imum amount -of time in which pupils should attain the objec- 

' tives for which the teacher is to be held accountable. 



Requirement of regular .reporting of pupil attainment of ob- , 
jectives to a person (in tthis case, the person was the school 
principal) to whom the .teacher is accountable. 

Each accountability factor was operationalized using materials -and pro- 
cedures that coiild" be developed cooperatively by teachers and other 
individuals with responsibilities for the content and outcomes of in- ' 
structibn at' a district, county, or state Level. ^ , 



The authors wish to express their appreciation to Mrjl^ Mar jorie 
Craig of the Sati Diego, Unified ^School District for her assistance 
throughout the study. " . 



The hypothesis investigated in this study was that there is:^a .\^^. 
positive relationship between level c^f teacher accountability 'and leanner 
achievement on the instructional objectives fo^ whicTi the teacher is • * 
held accountable. The dependent variable was *end-of-year reading * . 
achievement of • first-grade pupils on the inferred instructional objective's 
of the regular Grade. 1 reading .program in the school i^district in which 
the study was' conducted. The lowes,t of the four levels o*-*£eacher . ' ^ 
accountability in the study involved use of the regular reading program- 
without instructional ol^jectives or any supplementary ipforraati'Sn related 
to accolintability. For this level, no- special accountability :^actqrs were 
included. The highest level of accountability involved combined use 'of ^ 
all the following factors: the regular reading program accompanied by a 
set of instructional objectives -for the program, tests for regular assess- 
ment of learner attainment of^ the objectives , a pacing schedule indicating 
the number of weeks in which given sets of objectives should be mastered , 
aad the requirement that teachers siibmit to Cheir principals^a record 
of pupil achievement scores on each regular assessment exercise ad- 
ministered throughout the year. 
f 

. / - • ^ ■ " ■ . ■ ^ 'ys . ^ ■ _ 

\ ... ■ [ ^ >IETHOD 

Sub j eats 

The-^S^s -consisted of all first-grade pupils in 15 schools in a 
metropolitan southern California school district. The 15 schools were 
selected from .among 60. schools in the district on the basis of their 
similarity, on matching factors employed in 'the study aii^ described below. 
The^totai sample included 1889; children from 67 first-^rade classes. 

Materials^—' ^.'^ 

The basic instructional program used with all groups in the .study 
was a commercially published Grade. 1 reading program* that is; b^h the 
most widely used of the tfiree California state-adopted prin^ary^grade " 
reading programs and the basic program used ±fi the district in which the 
study was conducted'. The complete Grad^e 1 progf^irf includes Qver 600 
new wo.rds, plus content in areas related to worjd decoding; comprehension, 
and study skills.- The program materials contain neither an exf)licit 
statement of the instructional objectives of the program nor, a concise^ 
summary of the content and skills that child[ren are expected to learn. 

The basis for .the accountability procedures used in, three experiment 
treatments was a^'set of objectives-based assessment a^d remediation ma- 
terials designed to supplement the basic* instructional program in reading. 
This ^supplementary system was designed to enable the teachers to 
(1) identify explicitly the skills and related content that children 
should leairn from the basic program, (2) determine at regular assessment 
intervals whether the intended learning has occurred, and (3) provide^ 
remedial instruction on each skill that had not. been mas t^^ered by the 
conclusion of an assessment period. . , 



^ The initial -steps in design of the supplementary syst,em involved 
inferring of . the instructional objectives, .of the basic program and subse- 
quent! identification of /the inferred objectives judged to be important . 
enough that teachers should be held resROh^ible for their attainment. 
From, among a total of seven objectives inferred for the program, three 
were judged^ tq be critical to the development of: reading skills for . 
firat-grade children. These, three objectives were the ability to 
(1) read the new basic words in the program, (2) say the sounds of -.various, 
letters and letter combinations contained in the program and (3)'soundi^ 
out and read" new words c9mposed of the letters and letter combinations- ^, 

As a. basis for indicating clearly, to _th 
children should attain and for pro^viding for regular assessment and 
remediation, the Sbjectives and content .for the basic program were 
divided intQ 15 units containing relatively equal amounts of ndw 
content to be attained by the pupils. .For"^ each of the 15 units, the 
supplementary program contained a listing in the -teachers manual o| all 
content (i.g., new basic words, sounds, and decoding words) for each 
objective', a simple teacher-administered group test meas\iring' J)upil 
performance on each objective, a Class Record Sheet for recording each 
'pupil's achievement by objective on each test^ and brief remed'lation 
exercises , keyed to each objective and designed"*for use with pupils not 
attaining a specified criterion score on. one or more unit objectives , • . 
Thus, a classroom set of supplementary materials consisted of a teachers' 
manual for the supplementary system, 15^ t^sts (one per unit) for each 
child, 15 Class Record , Sheets , and .1^ setts of short remedial' exercises . 
for jeach child to' use only if his .unit-test . performance indicated- a. need- 
for remediation on one or more .objectives!. The total cost of the 
year-long supplementary system, which Wasldesigned to be hdth simple and* 
inexpensive, was less than^^ cents per pv^il- \ . ' 

Procedures | ^ . , 

' ' — . ' • " . ' 

- ^ Four levels of teacher accountability were designed as the experi- 
mental' treatments Thesfe levels^ o^ accountability are ^d'escribed in order 
below, from lowest ^to highest level, . ■ 

Level 1: Program Only .'' leachers at *the schopls under this condi- 
tion used only the state-aciopted' commercial reading program serving 
as the basic ins true tii^nal program in the study. .Since this group 
of teachers used their regular reading program without supplementa- 
tion or experimental Intervention- other thanrpost testing, it, seems 
safe to assume that they taught reading as they normally^ do. 

Level 2: Program plus Gbjectives-btfied" Assessment System (OAS); . 
' ' This condition involved use of th6 basic instructional program plus 
the supplementary systeiQ designed for use with it. Thus, in 
addition to theobasic instructional program, te^achers in this group 
were provided with- a jinit-by-iinit listing of the objectives, that 
th^ir pupils should attain, tests to determine' whether pupils had 
a attained the objectives, and" shprt 'remedial exercises keyed to the' 
unit objectives, . • . 



' Level 3; Program plus OAS, plus Pacing OR Reporting , This , 

condition included use of both. the basic instructional ^program 
" and the objectives-abased supplementary system, plus . one additional 
accountability factor. For teachers in three of tHe six schools 
in this treatment, -the added accountabilLty fdctor was a. pacing^ 

' . ^ schedule specifying for each of the 15 units the maximum number 
^ of weeks that should be required for pupils to attain all ob- 
jectives of the upit. The pacing schedule for the year was . 
developed to permit completion ^of ;the entire Grade 1 pfbgram. 
Instead of the pacing schedule*, the added: accountability factor 

■ . ,for teachers' in/ the "other three schools in this^ treatment was ■ 
a reporting requirement. Following each unit test ,^ ^^eacl^ teacher 
"under the reporting condition was Required to submit .tOjher 
principal? a copy of the. Class Record. Sheet showing the test 
score of ,each pupil on each objective for the unit'. (This .level'=^^ 
was considered \o fall .between Levels 2 and 4 because it included 
one additional factor --either pacing or reportitig--to Level 2, . 
but not both,, additional factors ^.n combination, as-in Level 4. 
Including of the pacing and reporting factors separately under 
; this condition ^permit ted 'analysis of their individual effects.) 

■ Level 4; Program plus OAS Pacing AND Reporting .' This condition ^ 
involved use of the basic instructional program with alp. three 
accountability factors: \the objectives-based supplementary 
sy'stem, the pacing schedule, and the reporting requirement." The 
Class Record Sheets used for recording unit test scores and 
reporting -them to principals for this group also contained the 
recommended pacing schedules fdr each unit, thus enabling the 
principal to quickly determine whether the rate of progress of 
\a group or class was consistent with the recommended rate.^ 

The school Served as the- experimental unit in the study. From 
among -the 60 available school? in the district in which the study was 
conducted, five Schools each were selected to represent low^ "middle 
and high achievement levels in reading. The particiilar schools at ^ 
each* achievement level were selected because of their similarity on 
(1) end-of-f irst-grade mean scores from 'the preceding school year on • 
fhe California state-maridated Cooipevktpfe Reading Test and (2) percen- 
tage of minority group studeni^s-. ' Randomized blocking procedures were 
employed to assign the 15 schools to treatments. From within each 
block of five schools, two schools were .a^ssigned to^the Prdgram plus 
OAS plus Pacing or 'Reporting condition (one school to Pacing, one to 
Reporting) and one school each to "the other three levels of account- 
ability. , Thus, a total of three sphools , one from each of the three 
levels of the blocking factor, were assigned to each treatment except • 
the "Pacing or Reporting treatment. Two schools at each blocking level 
or a total of six schools were assigned^tO"' the Pacing or ^Reporting 
treatment to permit analysis of possibly differential effects of the 
pacing and rep'Drting requitements . >> 



A 90-ininute teac^her- training ses^sion-^ for each experimental 
^treatment -(Levels 2, 3 and 4) was held at thie beginning of £he school 
year to** train each group of teachers in* the procedures that they were 
to use^.u The training sessions were^ arranged and co'nducted by a primary- 
grade supervisor ^rom the^ participating district, using materials and 
procedures provided by thl^ experimenters. Because accoi^ntability pro- 
cedures^ may be threatening to many teachers/ a concentrated effort^ 
was made to describe the' supplementary system anxi requirements in as 
pleasant and non-thre^atening a manner as possible, both in the written j . 
materials and in the training session. . ' 

Data Sources ' " . 

^ A 45-item oral-response criterion test waS developed to "assess end- 
of-year achievement on the three instructional objectives of the reading 
program (1 reading program words, 2 - saying sounds of letters and 
letter combinations^ and 3 -reading new words composed of*letters and ^ 
letter combinations from the program) . t)ne item per objective for each 
of the first 11 units ^f -'the pro*gram was randomly selected from the 
population^ of all'^ available items and included in the test. Since Units 
12-15 of the program included ''content only for the "reading program words" 
objecftive,*" the three randomly^selected items for each of these four units » 
assessed perf or^pance only on the prog'ram-words objective. Thus, theV 
test contained 26 items assessing performance on this objective and 11' 
Items each (one item for each of the first 11 units^ for the letter- 
sounds and wprd-Hecoding . pbj ectives.' 

The criterion test wa^individually administered 'by trained examine:^s 
during the next-to-last week of the gohool yeat to a sample of 24 children 
per school selected at random from among all 'first graders in each of tHe 
15 schools. The 360' posttested children, therefor*e> included a total of 
144 from the six Pacing or Reporting schools (72 for each condition) and 
a total of 72 children from each group of thre^* schools af^'each of the 
three remaining accountability levels. The KR-21 reliability coefficient, 
computed on the sample of 360 i^ost±ested S^s, was, .96 ,fqr the complete 
45-item posttest. KR-21 "coefficients for the subtes.ts assessing perform.-, 
ance on the three objectives ranged from .92 to .95. 

Data w.ere <also collected on the amount of the program completed at 
each accountability - level. At the time of posttesting, teachers werB 
asked to indicate t;he number of the unit in** the' reading program in which 
each tested child was' working. 'Each' teache'r was also- interviewefd 
briefly at this time to determine whether any contamination of experi- 
menital procedures had occurred across accountability levels. ' p 

Data An'alysis . • ■ wf>,^ i> • 

Posttest mean scores were tabulated for each treatment group 'on 
each objective. An L-test (Page, 1963) for four rankings (accDunta- 
bllity levels) by three observations . (subtest scores on each of*the 



three objectives) was performed to determine the relati'onship between 
level of teacher accountability and end-pf-yedr pupil achievement on . 
the instructional ob^a ctiv os th e reading program. . 



/ .RESULTS . . ' 

' - ■ . « 

The posttest mean scores. by treatment and objective are shown m 
Table 1. Analysis of the table reveals that without exception, pupil 
abhieve'ment increased on each objective with each increase- In the level 
df ^^teacher accountability/ Mean, percentage scores ou the complete post- 
test ranged from 55 percent for the program-only gtoup representing the 
lowest level' of accountability in- the , study to 68 percent fdi Ssat the 
highest leveh-of aocouritability--the Pacing and Reporting group. The 
L-test for four accountability levels_by three subtest observations 
revealed ^a significant mdnotonic relati'onship <L = ^90, y|)<. 001) between 
level of teacher accountability and' pupil achievement cm the instruc- • 
tional .objectives of the program, % . . , • 



Table "^l" 



Posttest Mean Scores by Accountability "Level and Objective 









Objective- 




» ■ 

"Total 
Test 
'(45) 


Accountability 
Level 


Words 
(23 items) 


Letter 
Sounds 
(11 items > 


Word- 
Decoding 
(11 items) 


1 -'Program only " < 
(Lowest level)' *- 


13.3 

58% 


6'.4 ■ 
58% 


5.1 

46% . 


24.8 
' 55% 


2 - Progri^ + Objectives- 
based Assessment 
System (OAS) 


^■ 14.1 
61%^. 


■ 6.9 

63%,. ^ 


6.0 

55% 

" \ 


27,0 
. "66% 


3 - Program + OAS + 

Pacing or Reporting , 


14'. 9 

65% . 


7.2 
65% ■ 


6.4 . 

58% 


28 .'B 
63% 


4 - Program + OAS + 

Pacing &^ Reporting 


16.1 - 


11%. 


^^'6.7 
61% • * 


\30.7 
68% 


Note: L = 90, p<.001 for 4 

c ^ 

** ' : 

< : 


rankings x '3 


observations. 

/ ' ■ 




•* 

4' • ' 

0 ■ ■ * 


8 

f 









Mean scores were also .computed and compared for the two sub-groups 
within the Program plus OAS plus Pacing or Reporting treatments The , ^ ' - 
difference in mean score on the total test between fes under the pacing 
condition and Ss under the reporting condition within this treatment 
was less than one raw-score point (facing x= 28.2, Reporting x = 2:8.2). 

The mean number of units completed out of the total number of 15 ' 
uni-ts in the reading program is shown below by accountability level: r 

^ Accountability Level Units Completed 

* 1 - Program only 9.0 (60% of total' program) 

2 - Program -f OAS ^ 10.9 (73%) 

3a - Program OAS + Reporting * 9.7 (65%) 

3b - Prograjn ^ OAS + Pacing3 io.8 (72%) . . ' 

4 - Program + OAS + Pacing r ^> ' 

> and Reporting 11.2 (75%) 

A3 shown above, the pro'gram completion rate ranged from 60% 'at the lowest^ 
accountability 'level to 75% at the highest level. Conditions which 
.included a pacing schedule (Lev^els 3b and 4) generally showed a. higher 
completion' rate than the other lei^els* . However, the general tr^d. towSrd 
decreasing completion rates with a decrease in accountability level was 
broken by the high completion rate at Level 2. ^The^high completion 
rate at 'this level was due to the f ac't^ t^tat subjects in the low-achievement 
school iil^ this group -completed an average of 11.6 units, the second highest 
number of Lgiits completed by any school in the study and more than tw6 ^ 
units higher thanNthe average number completed by the other low-achievement 
Schools. Interviews with the teacberd at this school revealed that their, 
principal .had obt^n^d and /established the pacing schedule used' at 
Levels 3* and 4 and was colleptingi reporting sheets. on a regular basis. 
Thus , this one Level 2 school v/as^ essentially under ' the I^evel 4 Eacing 
and Reporting condition. • ^ ' 

' ^ '* " ' DISCUg^ION - 

This study wa*s cOTjducted to investigate the relationship between 
level 'of teacher accountability and. the reading achievement of first- 
grade pupils, ^he data revealed a- significant moiiotonic relationship 
between accountability ''level and pupil achievement.. An increment of 
ei^d~of-year pupil achievement on the 'instructional objectives of the 
first-gra'de reading* program used in the study occurred ji^ith each 
increase in* level of teacher accountabilit5^. _ ; • • . 



^The'two conditions under Level 3 a'tei shown separately in this ^ 
listing JLo indicate the- difference, in program completion tate as- 
sociated with the pacing cbnditio,n as cbntrasted with the repbrting 
►condition. . • « , . ^ ^ 



On^ Important'' aspect-^of the accountability factojrs employed in the • 
study ^as 'the provision f Qr fredjVient en f oute assessment and reporting 
9f. learner performance on specific instructional objectives, Oft^n, 
pupil achievement data are systematically collected within a sc^iool" or . 
district only through use of standardized tests administered ^at* the 
b'egiKinitig and/or end of the scljool year. Norm-referencei data collected 
at such .infrequent intervals are of relatively little use^to' the teacher 
in making instructional decisions, that will increase the chances for 
attainment of the objectives for whit:h the teacher is held Accountable, 

'"Because of * the type of information sought from the experiment, the 
potential value of the objectives-based en route assessment and reporting 
procedures were not fully exploited in the present study. For experimen- 
tal purposes, the i\eporting cbndition was = intentionally limited' to 
regular collection of the reports in order to permit *a^alysis of the 
effects of reporting per se, and no assistaYice or .further .direttions 
designed to "yi^l^i improved performance were provided for teacherfe who 
submitted Clas^ Record Sheets indicating sub-standard pupil achievement 
and/or pacing. However, in actual operation of an effective account- 
ability system, the regular reports..can be us*^ed by supervisory: personnel i 
at the. school of district level to identify teachers who need help in* 
improving their instruction. Such ca' formative evaluation procedure used 
in conjunction with a teafcher accountability systetii should, result in 
•improved instruction and concomitant improvements ^n pupil achievement. 

The accountability factors identified and operationalized * as ^part 
of the present study included provision-of (1)- learner objectives and a 
regular means. for assessing their attainment, (2) a pacing schedule to 
encourage and help teachers plan for an acceptable rate of progrejss 
throjigh the instructional program, and (3) a procedure for regular 
reporting of en route learner achievement to schoql ^offitials. Data 

* from the study reveal a significant positive relationship between tKe 
presence, pf these factors and pupil achievement in first-grade' reading. 
It seems probable that, in tha operation of a well-designed accountability 

^system in the schools, the positive effects of the accoiint^bility factars^ 
from this study ^could be increased beyond the levels obtained In the study 

■ by incorporating intervention procedures designed to dmprove th^ P^^~ 
formance. of teachers having sub-standard- pacing and/or pupil achievement. 
Potentially then, application in the schools of the accountability factors 
idciiitif.ie'd and employed in this Study should yield positive results 
bc^th in impleipenting an effective teacher-accountability system and 
in improving pupil achievement,' ^ 
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